UKFin+: Can We Trust AI to Automate Regulatory Compliance?

Rigorous Evaluation for Responsible Deployment

Professor Barry Quinn
Ulster University Business School

Mr Abhishek Pramanick
Queen’s University Belfast

Dr Fearghal Kearney
Queen’s University Belfast

Professor Jesus Martinez Del Rincon
Queen’s University Belfast

2025-11-01

The Promise vs The Reality

The Burden

Financial services: £780 billion annually on compliance
Manual review: £200-500 per submission
FCA supervises: 50,000+ firms
Timeline: Weeks to months for complex rules

The AI Promise

Automated compliance checking
Instant regulatory interpretation
50-70% cost reduction projected
RegTech market: $7.9B → $45.8B by 2032

But Should We Trust It?

The Reality Check

Can AI accurately extract regulatory requirements?
Does AI know when it’s wrong?
Which regulations can we safely automate?

Financial services collectively spend roughly £780 billion per year on compliance activities—a figure that has grown steadily over the past decade (cf. regtech2023market?; Financial Conduct Authority 2022). AI is often sold as a solution to this cost pressure: automated compliance checking, near real-time regulatory interpretation, and 50–70% potential cost reductions in specific use cases (regtech2023market?). The RegTech market itself is projected to grow from $7.9 billion in 2022 to ~\$45.8 billion by 2032 (around 17% compound annual growth) (regtech2023market?), signalling just how much policy-makers, supervisors, and industry are banking on technology to make compliance more sustainable.

[Pause, let that sink in]

But a growing body of work on AI in high-stakes domains—medicine, law, and finance—shows systematic problems: models that appear confident whilst being wrong, and systems that perform well in controlled benchmarks but fail in messy real-world environments (e.g. myers2024fault?; Birhane 2021; surden2019artificial?). Recent research in medical AI from Johns Hopkins, for example, describes a "dual challenge" for clinicians: avoiding over‑reliance on flawed AI recommendations while also avoiding under‑reliance on genuinely helpful suggestions, often without clear guidance on when to trust algorithmic outputs (myers2024fault?).

[Build tension]

So the question isn’t whether AI can contribute to compliance processes—it clearly can. The question is: which aspects can be safely automated, which require human-AI collaboration, and which demand purely human professional judgement? That’s what we set out to understand for regulatory compliance.

[Transition to next slide]

The Research Question

Can AI Safely Automate Regulatory Compliance in Finance?

The AI and Law research community increasingly emphasises rigorous evaluation of real systems on actual legal text.

Our Investigation:

Accuracy: How well do different AI architectures extract structured rules?
- Symbolic knowledge graphs vs neural language models vs hybrids
Reliability: Can AI systems tell when they’re wrong?
- Do confidence scores correlate with actual correctness?
Deployment: Which regulations can we safely automate?
- Where does human professional judgement remain essential?

Our Test

6 AI architectures (symbolic, neural, hybrid)
Canonical evaluation set: 36 European Fund Classification (EFC) rules (held-out subset of the EFC corpus, used only for testing, not for building the knowledge base)
Shared EFC knowledge base: ~190 manually encoded EFC rules in GraphDB, used to build the symbolic KB and as the retrieval corpus for GraphRAG/Logic-LM (evaluation rules not included)
Systematic evaluation with reproducible methodology on real regulatory text (not simplified benchmarks)

The AI and Law research community increasingly emphasises that we need rigorous evaluation of real systems on actual legal text, not just laboratory benchmarks or vendor claims. Following this standard, we asked three interconnected questions.

[Pause, let questions register]

First: How accurate are different AI architectures? We tested symbolic knowledge graphs—systems with manually encoded rules and explicit semantics—versus neural language models that learn patterns from data, versus hybrid approaches that combine both. Second: Can AI systems tell when they’re wrong? Do confidence scores actually correlate with correctness? This is critical for deployment—if we’re going to route cases based on confidence, we need to know whether that’s safe. Third: Which regulations can we safely automate? Where does human professional judgement remain essential?

[Point to test details]

We tested six distinct AI architectures on a specific regulatory sub-domain—investment fund classification. Around 190 European Fund Classification (EFC) rules from European fund category brochures are manually encoded into our GraphDB knowledge base and used as the shared corpus for GraphRAG/Logic-LM. Within that domain, we define a canonical test set of 36 EFC rules that are held out as our evaluation subset and are never used to build or query the KB. In ongoing work, we also use 47 UK investment-fund rules from an analogous regulatory rulebook to test robustness—asking how well these architectures transfer within the same domain but across jurisdictions. This isn’t simplified laboratory text—it’s real regulatory complexity: ambiguity, cross-references, professional standards. Computational law scholars recognise that legal and regulatory text exhibits distinctive properties—specialised vocabulary, formal syntax, pervasive ambiguity—that resist straightforward automation (nllp2024workshop?; regnlp2025workshop?). Our evaluation tests whether AI can handle these challenges, in line with calls for rigorous, real-system evaluation in AI and Law (bex2024ai?; icail2023proceedings?; evaluation2013icail?).

[Transition]

Finding #1 – Structural Parity Surprise

Traditional Knowledge Graphs vs Modern AI: A Surprising Tie

Recent computational legal studies distinguish between “law-as-code” (formal symbolic representations) and “law-as-data” (statistical pattern learning).

The Architectural Landscape

The Surprise: GraphDB remains structurally strongest (≈88% GED similarity), but modern AI architectures are closer than many would expect (≈72–74%), so there is no simple ‘AI wins’ or ‘AI fails’ story.

But HOW They Achieve It Differs

GraphDB (Symbolic)

Strengths:

✅ Perfect terminology (94% lexical consistency)
✅ Explicit constraint representation
✅ Transparent reasoning

Limitations:

❌ ~190 manually encoded rules
❌ 10 months expert effort
❌ Expensive to extend

LLMs (Neural)

Strengths:

✅ Zero manual encoding
✅ 2 days implementation
✅ Flexible coverage

Limitations:

❌ Severe lexical drift (2-4% consistency—uses different terminology)
❌ Requires canonicalization (post-processing to align terms)
❌ Black-box reasoning

The Trade-Off

This isn’t about one architecture “winning”—it’s about fundamentally different trade-offs between knowledge engineering costs and output processing complexity. Your choice depends on operational context, not raw accuracy.

First major finding—and it’s a surprise. Recent computational legal studies distinguish between ‘law-as-code’—formal, executable representations of legal rules—and ‘law-as-data’—statistical pattern learning over large corpora (chau2024computational?; surden2019artificial?). We tested both paradigms on identical regulatory text, building on work that uses legal knowledge graphs for structured reasoning (kondo2025capturing?) while also engaging with the Natural Legal Language Processing literature (nllp2024workshop?).

[Point to architectural spectrum diagram]

The architectural landscape shows four systems spanning this spectrum. On the left, GraphDB represents law-as-code: 190 manually encoded rules, 10 months of expert effort, but perfect terminology alignment. On the right, Few-Shot LLM represents law-as-data: zero manual encoding, 2 days implementation, but severe lexical drift. In the middle, hybrid approaches like GraphRAG and Logic-LM try to combine both.

[Pause for effect]

The surprise: although GraphDB remains structurally strongest (≈88% GED similarity), the best neural and hybrid architectures reach ≈72–74% structural similarity—closer than many would anticipate—measured by graph edit distance (how similar the extracted regulatory logic structure is to ground truth), which assesses whether systems correctly capture regulatory dependencies and relationships. They get there through entirely different mechanisms with complementary failure modes.

[Point to trade-offs]

Symbolic systems front-load effort through manual encoding—expensive setup but clean outputs. Neural systems defer effort through post-processing—cheap setup but requires canonicalization. Neither dominates. Computational legal studies researchers have been calling for exactly this kind of fair comparison across paradigms, but rarely has anyone evaluated both on identical tasks with unified metrics—this is the first systematic evidence, on real regulatory text, of how symbolic and neural approaches differ structurally and where AI architectures begin to approach knowledge-graph performance.

[If asked about technical architectures]

GraphDB (Symbolic): Manually encoded RDF graphs for ~190 European fund classification rules; SPARQL queries over a fixed ontology; no learning, purely rule-based retrieval and reasoning.
Few-Shot LLM (Neural): GPT-4 style LLM prompted with a handful of annotated examples; maps raw rule text directly to RDF triples and constraint nodes in one shot, without retrieval or external KB.
GraphRAG (k=10 / k=50): Retrieval-augmented generation: embed rule text + SPARQL patterns, retrieve top-K similar EFC graphs from GraphDB, then feed those graphs plus the query text into the LLM to generate a new RDF graph.
Logic-LM (Hybrid): Two-stage pipeline: LLM proposes an RDF graph, then a logical validator checks constraints (e.g. percentage sums, category consistency); failed checks trigger repair or downgrade of reliability.
Constrained Generation (Symbolic-Neural): LLM decoding constrained by a schema/template for allowed predicates, node types, and constraint slots—reduces format errors but does not perform full logical validation.
Ambiguity-Aware Model: Same GraphRAG extraction pipeline, but paired with a separate ambiguity-scoring module over the rule text; routing decisions (auto / hybrid / human) depend on ambiguity score rather than model-reported confidence.

[Transition]

But here’s where it gets interesting—and concerning…

Finding #2 – The Confidence Trap

When AI Is Most Confident, It’s Often Most Wrong

Parallel from Medical AI

Recent research at Johns Hopkins shows doctors face similar challenges calibrating reliance on AI recommendations—but this is the first systematic documentation of inverse calibration in regulatory compliance automation.

Research on AI in high-stakes medical decisions has documented “overconfidence” problems. But we discovered something more concerning: systematic inverse calibration.

Expected vs Reality

Expected (Good Calibration):

High Confidence → High Accuracy ✅

When model says “90% confident,” accuracy should be ~90%

Reality (Inverse Calibration):

High Confidence → Low Accuracy ❌

When model says “90% confident,” accuracy is often worst

The Discovery: Negative Confidence-Accuracy Correlation

Model Architecture	Calibration (r)*	Interpretation
Few-Shot LLM	-0.545	🚨 INVERSE
GraphRAG (k=10)	-0.469	🚨 INVERSE
GraphRAG (k=50)	-0.500	🚨 INVERSE
Constrained Generation	-0.352	🚨 INVERSE
Logic-LM	+0.135	✅ Positive (weak)
Chain-of-Thought	+0.190	✅ Positive (weak)

*Correlation coefficient: +1 = perfect calibration, 0 = no relationship, -1 = inverse calibration

The Deployment Disaster

Naive confidence-based routing would:

✅ Automate the WORST predictions
- Model is confident → must be right!
❌ Flag CORRECT predictions for review
- Model uncertain → better check!
Result: Systematically route errors to automation

Example

Rule: “Fund must allocate ≥80% to bonds, ≥30% to equities…”

Model Output:

Confidence: 95% 😊
Accuracy: WRONG (80%+30%>100%)
Naive routing: AUTOMATE THIS! ☠️

Key Insight

This is the first systematic characterization of inverse calibration specifically in regulatory text extraction tasks. It has profound implications for safe deployment.

For AI to be safely deployed in regulatory compliance, it needs to know when it’s wrong. When practitioners see a confidence score, they need to know: can I trust this? Should I verify it? Must I override it? In high-stakes domains, reliable uncertainty estimates aren’t just nice to have—they’re essential for safe operation.

[PAUSE - establish importance]

Medical AI research has documented this challenge. Johns Hopkins researchers found that doctors integrating AI into clinical decision-making face what they call a ‘dual challenge’: avoid over-relying on flawed AI recommendations, but also avoid under-relying and rejecting genuinely helpful ones. The problem? They have to make these calibration judgements in real-time, under legal liability, without clear guidance on when AI outputs are trustworthy.

[Gesture to medical analogy on screen]

So we tested whether regulatory AI provides reliable confidence signals. What we expected to find—was positive correlation. When the model says ‘I’m 90% confident,’ accuracy should be around 90%. That’s what calibration means.

[PAUSE]

What we actually found—was the opposite. Inverse calibration. When confidence goes UP, accuracy goes DOWN.

[PAUSE FOR IMPACT]

Four out of six AI models exhibit NEGATIVE correlation between confidence and correctness. They are systematically most confident when most wrong.

[Point to data table]

Look at the numbers. Few-Shot LLM: correlation of negative 0.545. GraphRAG: negative 0.469 to negative 0.50. And here’s what’s particularly troubling—retrieval-augmented generation, which is supposed to improve performance by providing relevant context, actually makes calibration WORSE. More context inflates confidence beyond what improved accuracy warrants. Remember this sits on top of the structural picture: symbolic GraphDB reaches about 88% GED similarity, whilst the best AI architectures cluster around 72–74%.

[Point to bottom two rows]

Only two architectures achieve positive calibration: Logic-LM and Chain-of-Thought. Both force models to articulate reasoning before generating outputs. When reasoning hits difficulties, genuine uncertainty emerges. But even these correlations are weak—under 0.2.

[Point to code snippet]

Here’s the disaster scenario. An organisation deploys confidence-based routing. Seems reasonable: if the model is highly confident, automate; if it’s uncertain, get human review.

[Point to WRONG result]

With inverse calibration, this systematically routes your WORST predictions to automation. The model says ‘I’m 95% confident’—you automate it—and it’s completely wrong.

[Point to real-world example]

Here’s an actual case from our evaluation. The rule specifies constraints that sum to 120%—logically impossible. A human expert would spot this immediately. The model extracts them faithfully, reports 95% confidence, and naive routing says: automate this.

[Let that sink in]

You’re not just losing efficiency gains. You’re automating errors whilst wasting resources reviewing correct outputs. It’s the worst possible outcome.

[Gesture to key insight box]

To our knowledge, this is the first systematic characterization of inverse calibration specifically in regulatory text extraction. Medical AI research warned about overconfidence, but we’ve documented something worse: a systematic negative relationship between confidence and correctness in RegTech applications.

[PAUSE]

This finding changes everything about how we deploy these systems. We can’t use confidence scores for routing decisions. We need an alternative signal—something measurable from the regulatory text itself rather than unreliable model self-assessment.

[Transition]

Which brings us to our third finding about ambiguity…

Finding #3 – The Ambiguity Reality

Ambiguity Isn’t the Exception—It’s How Regulation Works

Legal scholars have long recognised that ambiguity serves essential functions—permitting contextual application, enabling evolutionary interpretation, and preserving professional judgement (esposito2021transparency?; he2025statutory?; dugac2025classifying?).

Analysis of 34 FCA Regulations:

pie title "FCA Regulations by Ambiguity"
    "Ambiguous Terms (60%)" : 60
    "Purely Quantitative (40%)" : 40

60% Ambiguous Terms
40% Purely Quantitative

The 40% We Can Automate:

“≥80% of fund assets must be equities”
“Net asset value calculated daily”
Clear numerical thresholds

The 60% That Resist Automation:

“Reasonable steps to verify…”
“Appropriate controls for risk…”
“Material changes disclosed…”
“Predominantly invested in…”

This Isn’t Bad Drafting

This Is Essential Design

These ambiguous terms serve critical regulatory functions:

Flexibility across diverse business models
Adaptability as markets evolve
Professional judgement for contextually appropriate application

Recent computational law research argues that attempting to eliminate ambiguity through rigid formalization would either produce unworkable specificity or drain rules of practical meaning.

Regulation requires some ambiguity.

Third finding challenges a common assumption. Legal theorists have long argued that ambiguity in legal language serves essential functions—permitting contextual application, allowing evolutionary interpretation, and maintaining space for professional judgement (esposito2021transparency?; he2025statutory?). Our empirical analysis of 34 FCA rules, showing that 60% contain such terms, provides quantitative support for this jurisprudential insight.

[Point to pie chart]

Analysis of 34 FCA regulations shows 60% contain ambiguous terms requiring professional interpretation—words like ‘reasonable,’ ‘appropriate,’ ‘material,’ ‘predominantly.’ The other 40% are purely quantitative: clear numerical thresholds like ‘at least 80% of assets’ or ‘maximum 10% exposure.’

[Pause]

This isn’t regulatory failure—it’s how regulation accommodates diverse contexts and evolving markets. Terms like ‘reasonable,’ ‘appropriate,’ and ‘predominantly’ cannot be eliminated without breaking the regulatory system. Legal theorists have long argued that ambiguity in regulatory language isn’t a bug but a feature—it permits contextual application, allows regulations to evolve with markets, and preserves space for professional judgement where rigid rules would break.

[Point to callout box]

These ambiguous terms serve critical regulatory functions: flexibility across diverse business models, adaptability as markets evolve, and professional judgement for contextually appropriate application. Recent computational law research argues that attempting to eliminate ambiguity through rigid formalization would either produce unworkable specificity or drain rules of practical meaning. Regulation requires some ambiguity.

[Transition]

So if ambiguity is fundamental and calibration is broken, how do we deploy safely?

The Solution – Graduated Automation

Route by Rule Ambiguity, Not Model Confidence

Given inverse calibration (confidence scores mislead) and pervasive ambiguity (60% of rules resist full automation), we developed an ambiguity-aware routing framework.

Why Ambiguity Matters

Legal Ambiguity Serves Essential Functions

Legal scholars recognise ambiguity isn’t bad drafting—it’s essential design:

Permits contextual application across diverse business models
Allows evolutionary interpretation as markets change
Preserves professional judgement for contextually appropriate application

We don’t try to eliminate ambiguity—we design around it.

Example: “Fund must hold ≥80% equity securities”

Characteristics: Purely quantitative thresholds, clear definitions
Processing: AI extraction → automated validation → no human review
Coverage: ~40% of FCA rules
Risk: Low (clear requirements, verifiable outcomes)

Example: “Fund predominantly invested in equity securities”

Characteristics: Partial specification, requires interpretation
Processing: AI extraction → human verification → expert approval
Coverage: ~20% of FCA rules
Risk: Medium (AI suggests, human validates)

Example: “Fund employs appropriate controls for liquidity risk”

Characteristics: Professional standards, firm-specific context
Processing: Human interpretation → expert encoding → peer review
Coverage: ~40% of FCA rules
Risk: High (requires professional judgement)

Why This Works

❌ Traditional (Fails):

IF model.confidence > 0.8:
    automate()  
    # ⚠️ Routes worst predictions!
ELSE:
    human_review()

✅ Our Approach (Robust):

ambiguity = measure_text(text)

IF ambiguity < 0.3:
    automate()  # Safe
ELIF ambiguity < 0.6:
    hybrid_processing()
ELSE:
    human_review()  # Preserve judgement

Key Advantage

Doesn’t rely on unreliable AI self-assessment. Routes based on measurable properties of regulatory text that we can verify independently.

After discovering inverse calibration, we needed a routing mechanism that doesn’t depend on broken confidence scores. But we also needed to step back and think about what we’re actually trying to automate. Legal scholars have long recognised that ambiguity in regulatory language serves essential purposes—it lets rules adapt to different contexts, permits evolutionary interpretation as markets change, and preserves space for professional judgement (esposito2021transparency?; he2025statutory?).

[Gesture to opening frame]

So we measured ambiguity empirically. Analysis of 34 FCA regulations found that 60% contain ambiguous terms requiring interpretation—words like ‘reasonable,’ ‘appropriate,’ ‘material,’ ‘predominantly.’ These aren’t drafting errors. They’re how regulation works across diverse firms and evolving markets.

[Point to pie chart]

[PAUSE]

This tells us one-size-fits-all automation can’t work. Different regulations have fundamentally different automation potential.

[Point to green tier]

Low ambiguity regulations—purely quantitative thresholds like ‘at least 80% equity holdings’—can be safely automated. No interpretation needed, just computational verification. This covers about 40% of FCA rules.

[Point to yellow tier]

Moderate ambiguity—terms like ‘predominantly invested’—need hybrid processing. AI extracts a suggested interpretation, but a compliance officer verifies it’s appropriate for this context. Quick human check, not full manual encoding. About 20% of rules.

[Point to red tier]

High ambiguity provisions—professional standards like ‘appropriate controls’—require human review. These invoke firm-specific context and regulatory judgement that AI shouldn’t and can’t handle reliably. About 40% of rules, and that’s okay. We’re not trying to automate everything.

[Point to code comparison]

The key architectural decision: we route by measurable text properties, not unreliable model confidence. Ambiguity scores come from the regulatory text itself—hedging language, subjective terms, professional standards—properties we can verify independently of any AI model.

[Gesture emphatically]

This makes the system robust to the calibration failures we documented. Doesn’t matter if the model is overconfident—we’re not asking it to assess its own reliability. It also lets us safely deploy models that are structurally weaker than our GraphDB baseline—around 72–74% GED versus roughly 88%—because we are not relying on their self-assessment to decide what gets automated.

This isn’t just pragmatic engineering—it’s grounded in how legal language actually functions. We’re respecting fundamental properties of regulatory communication rather than pretending better AI will eliminate ambiguity. Some rules demand human judgement. System design should acknowledge that, not ignore it.

Recent computational law research on statutory interpretation argues that attempting to eliminate ambiguity through precise formalization either produces unworkable specificity or drains rules of their practical meaning—our graduated automation framework respects this insight by measuring ambiguity from the text itself and routing accordingly, rather than pretending AI will eventually handle everything.

Impact & Deployment

Real-World Efficiency Gains With Safety Maintained

Projected Performance (1,000 regulatory submissions):

Tier	Rules	Processing	Cost/Sub	Human Effort
Automatic	40%	AI only	$1	0%
Hybrid	20%	AI + verification	$50	10-20 min
Human	40%	Expert review	$200	Full effort
CURRENT	100%	Manual	$300	100%
PROPOSED	100%	Graduated	$110	~40%

Cost-Benefit & Safety

Per Institution (10,000 checks/year):

Current cost: ~$3M annually
Projected cost: ~$1.1M annually
Projected savings: ~$1.9M/year

UK Financial Sector (1,000+ firms):

Sector-wide efficiency: hundreds of millions annually
(Caveat: Requires industry pilot validation)

Safety Properties:

✅ High-risk ambiguous rules get human review

✅ Quality maintained through graduated oversight

✅ Transparent routing rationale

✅ Robust to calibration failures

Meaningful efficiency gains—roughly 60% cost reduction—whilst maintaining safety. Let me show you what that means in practice.

[Point to table]

For 1,000 regulatory submissions, current manual processing costs about $300 per submission—that’s $300,000 total. With graduated automation, we automate the 40% that are purely quantitative—those cost $1 each. The 20% moderate-ambiguity cases get hybrid processing—AI drafts, human verifies—costing about $50 each. The 40% high-ambiguity cases stay fully manual at $200 each. Total: about $110,000—roughly 60% reduction.

[Point to cost-benefit]

For a mid-sized institution doing 10,000 regulatory checks annually, that’s roughly $1.9 million in projected savings per year. Across the UK financial sector with 1,000+ supervised firms, the efficiency potential is hundreds of millions annually—though this requires industry pilot validation.

[Point to safety properties]

Critically, safety is maintained. High-risk ambiguous rules get human review. Quality is maintained through graduated oversight—we’re not replacing human judgement, we’re focusing it where it matters most. The routing rationale is transparent—based on measurable text properties—and the system is robust to the calibration failures we documented.

Most legal AI research focuses on reporting accuracy metrics without addressing operational deployment—when to trust systems, how to integrate oversight, what happens when things go wrong—but we’ve translated rigorous empirical evaluation directly into operational guidance that practitioners can actually use, bridging what researchers call the ‘evaluation-to-deployment gap’ in high-stakes AI applications.

[Transition]

What Makes This Different

The Contributions

EMPIRICAL DISCOVERY

The Confidence Trap Documented

First quantitative evidence of inverse calibration in RegTech
4 of 6 architectures: r < -0.35
Models systematically most confident when most wrong
Implication: Naive confidence-based routing would be dangerous

Medical AI shows similar problems but not for regulatory compliance

METHODOLOGICAL RIGOR

Fair Symbolic-Neural Comparison

Unified evaluation framework enabling direct comparison
Graph edit distance (structural similarity) after IRI normalization (standardizing identifiers)
Structural parity finding: Both paradigms achieve ~73% similarity
Documents complementary trade-offs rather than dominance

Symbolic ≈ Neural (both ~73% accurate) with different strengths

PRACTICAL FRAMEWORK

Graduated Automation That Works

Ambiguity-aware routing robust to calibration failures
Routes by measurable text ambiguity, not unreliable confidence
Operationalizes legal theory: ambiguity serves essential purposes
Empirically grounded thresholds (based on FCA rule analysis)

Route by ambiguity not confidence scores

So what makes this work different from prior research? Three contributions that haven’t been done together before—empirical discovery, methodological rigor, and practical frameworks that actually work despite the problems we found.

[Point to first column]

First, we documented inverse calibration systematically for the first time in RegTech. Medical AI research warned about overconfidence generally, but we found something worse: four out of six models show negative correlation between confidence and accuracy. Models are systematically most confident when most wrong. That’s not just a technical curiosity—it’s a deployment safety crisis. And we found the retrieval paradox: more context improves accuracy but makes calibration worse. These are novel empirical findings specific to regulatory compliance automation.

[Point to second column]

Second, we enabled fair comparison across fundamentally different AI paradigms. Legal knowledge graphs and large language models usually get evaluated separately on different datasets with different metrics. We used unified evaluation—same 36 rules, same metrics, same preprocessing—and saw a clear structural lead for GraphDB (≈88% GED similarity) with the best neural/hybrid models clustered around 72–74%. The choice is still about trade-offs rather than simple dominance: symbolic systems require expensive manual encoding but produce clean outputs; neural systems need zero encoding but suffer lexical drift requiring post-processing. Symbolic systems require expensive manual encoding but produce clean outputs. Neural systems need zero encoding but suffer lexical drift requiring post-processing. The finding isn’t that one dominates; it’s that system choice depends on your operational context—available expertise, coverage needs, ontology alignment requirements. That’s methodologically novel: comparing paradigms fairly enough to reveal real trade-offs.

[Point to third column]

Third, we developed a deployment framework that works despite broken calibration. Graduated automation routes by measurable text ambiguity—quantitative terms, hedging language, professional standards—not unreliable model confidence. This is grounded in both empirical analysis—60% of FCA rules contain ambiguous terms—and legal theory that recognises ambiguity serves essential regulatory functions. We’re not waiting for perfect AI or pretending confidence scores will get better. We’re designing around documented limitations whilst still extracting meaningful automation gains.

[Gesture across all three columns]

The distinctive part isn’t any single contribution in isolation. It’s the systematic integration: rigorous empirical evaluation revealing problems, fair methodological comparison showing trade-offs, and practical frameworks that acknowledge limitations whilst enabling safe deployment. That combination moves from ‘here’s what AI can do in principle’ to ‘here’s how to deploy it safely in practice.’

[Brief pause]

So what does this mean for anyone trying to deploy AI in compliance?

Key Takeaways

Three Essential Points for Anyone Deploying AI in Compliance

The Three Takeaways

1. 🚨 AI Can’t Reliably Tell When It’s Wrong

4 of 6 architectures inversely calibrated (r < -0.35)
Models most confident when most wrong
Don’t trust confidence scores for routing decisions

2. 📊 Ambiguity Is Fundamental, Not Fixable

60% of FCA regulations contain ambiguous terms
Not regulatory failure—essential for flexibility
Design around ambiguity rather than assuming better AI will eliminate it

3. ✅ Graduated Automation Works Despite Limitations

Route by measurable text ambiguity, not unreliable confidence
~60% projected cost reduction with safety maintained
First deployment framework addressing inverse calibration problem

Three takeaways for anyone deploying AI in compliance, consistent with emerging standards for responsible AI in legal domains (bex2024ai?; surden2019artificial?).

[Point to first takeaway]

First, AI can’t reliably tell when it’s wrong. Four of six architectures are inversely calibrated—correlation coefficients less than negative 0.35. Models are systematically most confident when most wrong. Don’t trust confidence scores for routing decisions. This is the first systematic documentation of this phenomenon in regulatory text extraction.

[Point to second takeaway]

Second, ambiguity is fundamental, not fixable. 60% of FCA regulations contain ambiguous terms. This isn’t regulatory failure—it’s essential for flexibility and context-sensitivity. Recent computational law research confirms ambiguity serves critical legal functions. Design around ambiguity rather than assuming better AI will eliminate it.

[Point to third takeaway]

Third, graduated automation works despite these limitations. Route by measurable text ambiguity, not unreliable confidence. We project roughly 60% cost reduction with safety maintained. This is the first deployment framework addressing the inverse calibration problem we documented.

[Pause]

The broader lesson: rigorous evaluation revealing limitations enables better deployment than optimistic assumptions that ignore them. This research demonstrates how going beyond accuracy to include calibration, architectural comparison, and ambiguity analysis enables safe deployment of AI in high-stakes regulatory contexts despite systematic limitations. The path forward isn’t waiting for perfect AI but designing thoughtful automation that acknowledges capabilities and constraints.

[Transition to Q&A]

Thank You + Questions

UKFin+: Rigorous Evaluation for Responsible Deployment

Key Achievement:

Systematic empirical assessment of AI for regulatory compliance that documents both capabilities and critical limitations, enabling graduated automation despite inverse calibration.

Contact:

Professor Barry Quinn CStat, PhD
b.quinn@ulster.ac.uk

Resources:

Full evaluation methodology
Graduated automation implementation
Calibration analysis tools
Ambiguity measurement framework

Next Steps:

Industry pilots with financial institutions
Extended evaluation across regulatory domains
Open-source release of evaluation framework
Academic publication in computational law venues

Discussion Topics:

Inverse calibration in your domain?
Ambiguity profile of your regulations?
Architectural trade-offs for your context?
Workflow integration challenges?

UKFin+: Can We Trust AI to Automate Regulatory Compliance?

The Promise vs The Reality

The Burden

The AI Promise

But Should We Trust It?

The Research Question

Can AI Safely Automate Regulatory Compliance in Finance?

Finding #1 – Structural Parity Surprise

Traditional Knowledge Graphs vs Modern AI: A Surprising Tie

The Architectural Landscape

But HOW They Achieve It Differs

GraphDB (Symbolic)

LLMs (Neural)

Finding #2 – The Confidence Trap

When AI Is Most Confident, It’s Often Most Wrong

Expected vs Reality

The Deployment Disaster

Finding #3 – The Ambiguity Reality

Ambiguity Isn’t the Exception—It’s How Regulation Works

This Isn’t Bad Drafting

The Solution – Graduated Automation

Route by Rule Ambiguity, Not Model Confidence

Why Ambiguity Matters

Three-Tier System

Why This Works

Impact & Deployment

Real-World Efficiency Gains With Safety Maintained

Cost-Benefit & Safety

What Makes This Different

The Contributions

EMPIRICAL DISCOVERY

METHODOLOGICAL RIGOR

PRACTICAL FRAMEWORK

Key Takeaways

Three Essential Points for Anyone Deploying AI in Compliance

The Three Takeaways

1. 🚨 AI Can’t Reliably Tell When It’s Wrong

2. 📊 Ambiguity Is Fundamental, Not Fixable

3. ✅ Graduated Automation Works Despite Limitations

Thank You + Questions

UKFin+: Rigorous Evaluation for Responsible Deployment

Questions?

Backup: Model Architectures

Technical Structure of Each Model (for Q&A)

Thank you for your attention